Search

Stream processing has been an active research field for more than 20 years, but it is now witnessing its prime time due to recent successful efforts by the research community and numerous worldwide open-source communities. This survey provides a comprehensive overview of fundamental aspects of stream processing systems and their evolution in the functional areas of out-of-order data management, state management, fault tolerance, high availability, load management, elasticity, and reconfiguration. We review noteworthy past research findings, outline the similarities and differences between the first ('00–'10) and second ('11–'23) generation of stream processing systems, and discuss future trends and open problems. [ABSTRACT FROM AUTHOR]

Logistics 4.0 aims at enabling the sustainable satisfaction of customer demands with optimised costs of services with the use of emerging technologies, such as Internet of Things, streaming analytics, and optimised decision making. The availability of massive sensor data streams over time opens new perspectives for extracting meaningful and timely insights from data-in-motion through streaming analytics. Logistics 4.0 is a relatively new field of research which demands the development of scalable and efficient software solutions and their deployment to successful real-life case studies. In this paper, we propose a software framework for streaming analytics in an edge-cloud computational environment aiming at covering the whole data analytics lifecycle in logistics processes and thus, advancing the evolution and realisation of the Logistics 4.0 concept. The proposed framework takes advantage of edge computing technologies, streaming analytics and proactive decision making in order to monitor, analyse and support decision making in the frame of Logistics 4.0. It is applied and evaluated in a maintenance service logistics use case from the aerospace industry. [ABSTRACT FROM AUTHOR]

This report provides an introduction to the Computational Intelligence and Forecasting Technologies (CIFT) project at Lawrence Berkeley National Laboratory (LBNL). The main objective of CIFT is to promote the use of high-performance computing (HPC) tools and techniques for analysis of streaming data. After noticing the data volume being given as the explanation for the five-month delay for SEC and CFTC to issue their report on the 2010 Flash Crash, LBNL started the CIFT project to apply HPC technologies to manage and analyze financial data. Making timely decisions with streaming data is a requirement for many different applications, such as avoiding impending failure in the electric power grid or a liquidity crisis in financial markets. In all these cases, the HPC tools are well suited in handling the complex data dependencies and providing timely solutions. Over the years, CIFT has worked on a number of different forms of streaming data, including those from vehicle traffic, electric power grid, electricity usage, and so on. The following sections explain the key features of HPC systems, introduce a few special tools used on these systems, and provide examples of streaming data analyses using these HPC tools.

This report provides an introduction to the Computational Intelligence and Forecasting Technologies (CIFT) project at Lawrence Berkeley National Laboratory (LBNL). The main objective of CIFT is to promote the use of high-performance computing (HPC) tools and techniques for analysis of streaming data. After noticing the data volume being given as the explanation for the five-month delay for SEC and CFTC to issue their report on the 2010 Flash Crash, LBNL started the CIFT project to apply HPC technologies to manage and analyze financial data. Making timely decisions with streaming data is a requirement for many different applications, such as avoiding impending failure in the electric power grid or a liquidity crisis in financial markets. In all these cases, the HPC tools are well suited in handling the complex data dependencies and providing timely solutions. Over the years, CIFT has worked on a number of different forms of streaming data, including those from vehicle traffic, electric power grid, electricity usage, and so on. The following sections explain the key features of HPC systems, introduce a few special tools used on these systems, and provide examples of streaming data analyses using these HPC tools.

This work describes a novel end-to-end data ingestion and runtime processing pipeline, which is a core part of a technical solution aiming to monitor frailty indices of patients during and after treatment and improve their quality of life. The focus of this work is on the technical architectural details and the functionalities provided, which have been developed in a manner that are extensible, scalable and fault-tolerant by design. Extensibility refers to both data sources and the exact specification of analysis techniques. Our platform can combine data not only from multiple sensor types but also from electronic health records. Also, the analysis component can process the patient data both individually and in combination with other patients, while exploiting both cloud and edge resources. We have shown concrete examples of advanced analytics and evaluated the scalability of the system, which has been fully prototyped. [ABSTRACT FROM AUTHOR]

In the stream processing domain, applications are represented by graphs of operators arbitrarily connected and filled with their business logic code. The APIs of existing Stream Processing Systems (SPSs) ease the development of transformations that recur in the streaming practice (e.g., filtering, aggregation and joins). In contrast, their parallelism abstractions are quite limited since they provide support to stateless operators only, or when the state is organized in a set of key-value pairs. This paper presents how the parallel patterns methodology can be revisited for sliding-window streaming analytics. Our vision fosters a design process of the application as composition and nesting of ready-to-use patterns provided through a C++17 fluent interface. Our prototype implements the run-time system of the patterns in the FastFlow parallel library expressing thread-based parallelism. The experimental analysis shows interesting outcomes. First, our pattern-based approach allows easy prototyping of different versions of the application, and the programmer can leverage nesting of patterns to increase performance (up to 37% in one of the two considered test-bed cases). Second, our FastFlow implementation outperforms (three times faster) the handmade porting of our patterns in popular JVM-based SPSs. Finally, in the concluding part of this paper, we explore the use of a task-based run-time system, by deriving interesting insights into how to make our patterns library suitable for multi backends.

Real-time streaming technology and analytics capabilities are growing rapidly, whereas a great number of firms and organizations are considering implementing this technology to meet rising business demands. Traditional computer infrastructures for high performance computing and big data analytics are not able to conduct such tasks. To tackle this obstacle, rapid analysis of streaming data requires significant amounts of computer and data storage capacity, which requires real-time streaming technology and analytics. Real-time streaming has become a crucial component where tremendous volumes of data from thousands of sensors and other information sources are processed so that a company extracting the copious amount of real-time data can react to changing conditions instantaneously. Streaming technology and analytics generate real value from real-time data. This paper presents the current architecture, status, and trend of real-time streaming technology and analytics. It discusses value creation of streaming analytics. The paper describes continuous intelligence and value for streaming analytics and the current architecture and status of streaming technology and analytics; showcases the leading vendors for streaming technology and analytics; discusses various real-world use cases and benefits across various industries; analyzes value creation of streaming analytics; and proposes several research issues, along with challenges and recommendations. [ABSTRACT FROM AUTHOR]

In cloud computing, load balancing, energy utilization are the critical problems solved by virtual machine (VM) migration. Live migration is the live movement of VMs from an overloaded/underloaded physical machine to a suitable one. During this process, transferring large disk image files take more time, hence more migration and down time. In the proposed adaptive deduplication, based on the image file size, the file undergoes both fixed, variable length deduplication processes. The significance of this paper is resumption of VMs with reunited deduplicated disk image files. The performance measured by calculating the percentage reduction of VM image size after deduplication, the time taken to migrate the deduplicated file and the time taken for each VM to resume after the migration. The results show that 83%, 89.76% reduction overall image size and migration time respectively. For a deduplication ratio of 92%, it takes an overall time of 3.52 minutes, 7% reduction in resumption time, compared with the time taken for the total QCOW2 files with original size. For VMDK files the resumption time reduced by a maximum 17% (7.63 mins) compared with that of for original files. [ABSTRACT FROM AUTHOR]

Exploring Internet of Things (IoT) data streams generated by smart cities means not only transforming data into better business decisions in a timely way but also generating long-term location intelligence for developing new forms of urban governance and organization policies. This paper proposes a new architecture based on the edge-fog-cloud continuum to analyze IoT data streams for delivering data-driven insights in a smart parking scenario.

A class of efficient processing methods known as Streaming Analytics is emerging within Distributed Systems to analyze the contents of high velocity, high volume data streams in real time. Streaming Analytics can potentially solve a variety of computationally hard problems. However, realizing these solutions depends on the ability to feed high-throughput test streams into analytics systems under development. This article describes an exploratory effort to address this need by developing a tool called "StreamServer." The StreamServer team started by rehosting capabilities from SimServer, an existing web-based streaming tool, in a streamlined software architecture. This new application supported up to 25 times greater data throughput than the original SimServer. The proposed architecture integrated this application with Apache Kafka, a popular open source distributed messaging framework. The resulting StreamServer is a highperformance stream generator well suited to replicate and replay real world data streams for developing and testing Streaming Analytics systems. [ABSTRACT FROM AUTHOR]

The expansion of the electronic commerce, together with an increasing confidence of customers in electronic payments, makes of fraud detection a critical factor. Detecting frauds in (nearly) real time setting demands the design and the implementation of scalable learning techniques able to ingest and analyse massive amounts of streaming data. Recent advances in analytics and the availability of open source solutions for Big Data storage and processing open new perspectives to the fraud detection field. In this paper we present a Scalable Real-time Fraud Finder (SCARFF) which integrates Big Data tools (Kafka, Spark and Cassandra) with a machine learning approach which deals with imbalance, nonstationarity and feedback latency. Experimental results on a massive dataset of real credit card transactions show that this framework is scalable, efficient and accurate over a big stream of transactions. [ABSTRACT FROM AUTHOR]

With emerging Network Functions Virtualization (NFV) and Software Defined Networking (SDN) paradigms in Network Management (NM), new network devices and features can immediately become available. Available network resources and services can be altered and optimized in real time to gain the maximum benefit. However, this requires real time analytics information sent to SDN controllers rather than traditional manual offine or batch analytic outputs which are delivered on hourly or monthly basis in NM Systems. As a result, real time stream analytics is becoming a critical element for NM. In this paper, we describe a anomaly detection analytic engine for NM system development. We describe the design principles, innovative algorithm design, architecture and implementation of the engine in relation to streaming data and mobile NM. Finally, we present use cases and evaluation results. [ABSTRACT FROM AUTHOR]

An increasing amount of information is collected in industrial production processes. In many cases, this data is only accessible to the vendor of the machines involved in the production process. In the government funded research project BigPro, we propose a flexible and fast Big Data platform that allows detection and reaction to incidents and anomalies in the production process in near real-time. [ABSTRACT FROM AUTHOR]

We present a scalable system for high-throughput real-time analysis of heterogeneous data streams. Our architecture enables incremental development of models for predictive analytics and anomaly detection as data arrives into the system. In contrast with batch data-processing systems, such as Hadoop, that can have high latency, our architecture allows for ingest and analysis of data on the fly, thereby detecting and responding to anomalous behavior in near real time. This timeliness is important for applications such as insider threat, financial fraud, and network intrusions. We demonstrate an application of this system to the problem of detecting insider threats, namely, the misuse of an organization's resources by users of the system and present results of our experiments on a publicly available insider threat dataset. [ABSTRACT FROM PUBLISHER]

Extracting latent low-dimensional structure from high-dimensional data is of paramount importance in timely inference tasks encountered with “Big Data” analytics. However, increasingly noisy, heterogeneous, and incomplete datasets, as well as the need for real-time processing of streaming data, pose major challenges to this end. In this context, the present paper permeates benefits from rank minimization to scalable imputation of missing data, via tracking low-dimensional subspaces and unraveling latent (possibly multi-way) structure from incomplete streaming data. For low-rank matrix data, a subspace estimator is proposed based on an exponentially weighted least-squares criterion regularized with the nuclear norm. After recasting the nonseparable nuclear norm into a form amenable to online optimization, real-time algorithms with complementary strengths are developed, and their convergence is established under simplifying technical assumptions. In a stationary setting, the asymptotic estimates obtained offer the well-documented performance guarantees of the batch nuclear-norm regularized estimator. Under the same unifying framework, a novel online (adaptive) algorithm is developed to obtain multi-way decompositions of low-rank tensors with missing entries and perform imputation as a byproduct. Simulated tests with both synthetic as well as real Internet and cardiac magnetic resonance imagery (MRI) data confirm the efficacy of the proposed algorithms, and their superior performance relative to state-of-the-art alternatives. [ABSTRACT FROM PUBLISHER]

Exploring Internet of Things (IoT) data streams generated by smart cities means not only transforming data into better business decisions in a timely way but also generating long-term location intelligence for developing new forms of urban governance and organization policies. This paper proposes a new architecture based on the edge-fog-cloud continuum to analyze IoT data streams for delivering data-driven insights in a smart parking scenario. [ABSTRACT FROM AUTHOR]

Modern manufacturing processes can be monitored using a wide variety of sensors that can be used for tasks such as process control, predictive maintenance, and defect detection and prediction. It would be desirable within an Industry 4.0 framework to use machine learning techniques to both optimise data collection and develop pattern recognition. In order to accomplish this the application must be able to handle (near) real-time processing using streaming data obtained directly from the manufacturing process along with other appropriate sources. In order to update and evaluate the main streaming channel of the framework we propose a second longer term channel that can evaluate and optimise competing algorithms and data strategies to select the most appropriate for in-line detection or control. This paper looks at Lambda and Kappa architectures for Big Data analysis and methods of evaluating between competing in-stream machine learning techniques and data selection criteria as part of a dual channel Industry 4.0 intelligent system. The paper will examine criteria and metrics for selection and evaluation of suitable streaming learning algorithms along with architecture and strategy selections for live switching and modification of in-process control and monitoring processes. [ABSTRACT FROM AUTHOR]